nixos/binfmt: Add QEMU wrapper to preserve argv[0] by zhaofengli · Pull Request #143060 · NixOS/nixpkgs

zhaofengli · 2021-10-26T23:24:26Z

Motivation for this change

This PR enables argv[0] preservation for the QEMU binfmt-misc rules generated by boot.binfmt.emulatedSystems. It allows certain packages that rely on argv[0] in their build process (e.g., coreutils for its tests) to correctly build under binfmt.

Traditionally, the emulator invoked by binfmt-misc doesn't know what the original argv[0] is when running an executable, because only the absolute path to the foreign executable is passed. Currently, the emulator will simply use the full path as argv[0]. This leads to problems with multi-call executables as well as diff-based test methodologies used in various packages. For example, coreutils relies on output comparison in many test cases:

uniq: test 145.p: stderr mismatch, comparing 145.p.3 (expected) and 145.p.E (actual)
*** 145.p.3     Mon Oct 18 07:11:43 2021
--- 145.p.E     Mon Oct 18 07:11:44 2021
***************
*** 1,7 ****
! uniq: invalid argument 'badoption' for '--group'
  Valid arguments are:
    - 'prepend'
    - 'append'
    - 'separate'
    - 'both'
! Try 'uniq --help' for more information.
--- 1,7 ----
! /build/coreutils-8.32/src/uniq: invalid argument 'badoption' for '--group'
  Valid arguments are:
    - 'prepend'
    - 'append'
    - 'separate'
    - 'both'
! Try '/build/coreutils-8.32/src/uniq --help' for more information.

binfmt-misc now has the "P" flag that, when enabled, will add an extra argument after the absolute path to the foreign binary containing the original argv[0]. However, this also means that CLI compatibility with existing emulator executables is broken. SUSE has been carrying a patch since 2011 to add -binfmt wrappers that translate the new arguments into ones that regular QEMU accepts (-0 for overriding argv[0]). There have been attempts to resolve this issue upstream, but so far there seems to be no consensus.

This PR adds a statically-linked wrapper for QEMU that translates the arguments. It's possible to rewrite the arguments in the wrapper script that we use (#118926), but that still leaves some remaining issues (e.g., extraneous output when setting LD_PRELOAD).

Tested by building coreutils for riscv64-linux on x86_64-linux under binfmt (before, after).

Things done

oxalica · 2021-10-27T06:43:52Z

Dup of #118926

zhaofengli · 2021-10-27T06:51:29Z

Whoops, wasn't aware of that PR and GitHub wasn't returning good results for my searches. However, I still think the implementation here is more robust and only affects QEMU (not wine, for example).

zhaofengli · 2021-10-28T09:12:38Z

I implemented the suggestion from @oxalica in #115406 (comment) and coreutils can now be built under binfmt. I did a --check build and the results match what I got from native riscv64. The old version is here for comparison.

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.nix

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c

nixos/modules/system/boot/binfmt.nix

oxalica · 2021-10-28T12:22:39Z

Also we should update the PR title.

zhaofengli · 2021-10-31T04:24:26Z

Looks like extra-sandbox-path cannot contain the path to the executable itself, otherwise there will be some interesting conflicts when you try to build nix.conf.

To reproduce:

Activate emulatedSystems using the previous commit
After activation, run nix-build --check $(nix-store -qd /etc/nix/nix.conf) on the same machine

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c

oxalica

Just need more docs. It works pretty fine now.
I've started a build of riscv64 coreutils with binfmt from this PR. Hope it could pass tests.

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.nix

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c

oxalica · 2021-11-26T20:13:02Z

riscv64-linux coreutils built and passed all tests with QEMU wrapper from this PR now.

risicle · 2021-12-30T22:11:36Z

True to your word this does appear to fix the LD_PRELOAD error message with binfmt - would you mind adding a test such as the following for this?

  ldPreload = makeTest {
    name = "systemd-binfmt-ld-preload";
    machine = {
      boot.binfmt.emulatedSystems = [
        "aarch64-linux"
      ];
    };
    testScript = let
      helloAarch64 = pkgs.pkgsCross.aarch64-multiplatform.hello;
      libredirectAarch64 = pkgs.pkgsCross.aarch64-multiplatform.libredirect;
    in ''
      machine.start()

      assert "error" not in machine.succeed(
          "LD_PRELOAD='${libredirectAarch64}/lib/libredirect.so' ${helloAarch64}/bin/hello 2>&1"
      ).lower()
    '';
  };

risicle · 2021-12-30T22:15:53Z

@ofborg test systemd-binfmt

(I want to see what happens when this runs on an aarch64 machine)

…essages

risicle

Great work.

0x4A6F

Tests successfully run against rebased master.

LGTM

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Oct 26, 2021

zhaofengli mentioned this pull request Oct 26, 2021

stdenv: bootstrap riscv64-linux (Step 1) #115406

Merged

zhaofengli requested review from flokli, matthewbauer and shlevy October 27, 2021 02:14

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Oct 27, 2021

zhaofengli force-pushed the binfmt-argv0 branch from bc969b5 to aed1cf6 Compare October 28, 2021 09:01

ofborg bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Oct 28, 2021

oxalica suggested changes Oct 28, 2021

View reviewed changes

zhaofengli force-pushed the binfmt-argv0 branch from aed1cf6 to 8631fff Compare October 28, 2021 19:03

oxalica mentioned this pull request Oct 28, 2021

nixos/binfmt: correctly handle argv0 when preserveArgvZero #118926

Closed

10 tasks

zhaofengli force-pushed the binfmt-argv0 branch 2 times, most recently from 5c65b40 to 724d53c Compare October 28, 2021 19:28

zhaofengli changed the title ~~nixos/binfmt: Preserve argv[0] when using QEMU~~ nixos/binfmt: Add QEMU wrapper to preserve argv[0] Oct 29, 2021

zhaofengli force-pushed the binfmt-argv0 branch from 724d53c to 463294c Compare October 31, 2021 04:18

oxalica suggested changes Nov 2, 2021

View reviewed changes

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c Outdated Show resolved Hide resolved

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c Outdated Show resolved Hide resolved

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c Outdated Show resolved Hide resolved

zhaofengli force-pushed the binfmt-argv0 branch 2 times, most recently from c45d694 to 5d0aa2a Compare November 3, 2021 22:09

oxalica self-requested a review November 9, 2021 23:09

oxalica approved these changes Nov 26, 2021

View reviewed changes

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.nix Outdated Show resolved Hide resolved

pkgs/applications/virtualization/qemu/binfmt-p-wrapper.c Outdated Show resolved Hide resolved

oxalica mentioned this pull request Nov 29, 2021

stdenv: bootstrap riscv64 (Step 2) #147942

Merged

12 tasks

zhaofengli mentioned this pull request Nov 29, 2021

NixOS on RISC-V #101651

Closed

zhaofengli added 2 commits December 1, 2021 22:20

qemu: Add binfmt preserve-argv[0] wrapper

3011531

nixos/binfmt: Preserve argv[0] when using QEMU

9e5d0a9

zhaofengli force-pushed the binfmt-argv0 branch from 5d0aa2a to 9e5d0a9 Compare December 2, 2021 06:21

zhaofengli requested a review from dasJ as a code owner December 2, 2021 06:21

oxalica approved these changes Dec 3, 2021

View reviewed changes

zhaofengli mentioned this pull request Dec 30, 2021

libredirect fails to build on aarch64-darwin #152667

Closed

nixos/tests/systemd-binfmt: Add ldPreload test for LD_PRELOAD error m…

c8d1379

…essages

risicle approved these changes Dec 30, 2021

View reviewed changes

0x4A6F approved these changes Jan 6, 2022

View reviewed changes

0x4A6F merged commit 29acc14 into NixOS:master Jan 6, 2022

zhaofengli deleted the binfmt-argv0 branch January 6, 2022 19:17

queezle42 mentioned this pull request Jan 17, 2022

bison: Test failures while building with qemu-binfmt (e.g. armv7l-linux) #154308

Closed

This was referenced Feb 17, 2022

Cannot chroot to a different CPU architecture using boot.binfmt.emulatedSystems #160300

Closed

nixos/binfmt: Add support for using statically-linked QEMU #160802

Closed

Uh oh!

Conversation

zhaofengli commented Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for this change

Things done

Uh oh!

oxalica commented Oct 27, 2021

Uh oh!

zhaofengli commented Oct 27, 2021

Uh oh!

zhaofengli commented Oct 28, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oxalica commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaofengli commented Oct 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oxalica left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

oxalica commented Nov 26, 2021

Uh oh!

risicle commented Dec 30, 2021

Uh oh!

risicle commented Dec 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

risicle left a comment

Choose a reason for hiding this comment

Uh oh!

0x4A6F left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhaofengli commented Oct 26, 2021 •

edited

Loading

oxalica commented Oct 28, 2021 •

edited

Loading

zhaofengli commented Oct 31, 2021 •

edited

Loading

risicle commented Dec 30, 2021 •

edited

Loading